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ABSTRACT 

This article describes an effective approach to using the Enhanced Critical Incident 
Technique (EC1T) research method based on Flanagan’s (1954) Critical Incident Tech- 
nique (CIT). It begins with an overview of the CIT, how to decide if it is the appropriate 
methodology to use, then, using a recent CIT study as an example, discusses Flanagan’s 
five steps for conducting a CIT study: (a) determining the activity’s aim; (b) setting plans, 
specifications, and criteria for the information to be obtained; (c) collecting data; (d) 
analyzing the data; and (e) reporting the findings. Nine credibility checks, developed to 
increase the rigour and credibility of the ECIT, are described. 

RESUME 

Le present article decrit une methodologie efficace pour utiliser la methode de recherche 
de la technique amelioree des incidents critiques, Enhanced Critical Incident Technique 
(ECIT), basee sur la technique des incidents critiques, Critical Incident Technique (CIT) de 
Flanagan (1954). L’article commence par un survol de la CIT, puis passe a une determina- 
tion de l’adaptation de cette methodologie. Ensuite, se servant d’une etude CIT recente 
comme exemple, l’article discute des cinq etapes decrites par Flanagan pour mener une 
etude de CIT : (a) determiner l’objectif de l’activite; (b) etablir des plans, specifications, 
et criteres relatifs a l’information a obtenir; (c) collecter les donnees; (d) analyser les 
donnees; et (e) communiquer les conclusions. Neuf verifications de credibilite, elaborees 
pour augmenter la rigueur et la credibilite de la ECIT, sont decrites. 

This article chronicles the steps and processes involved in applying the Enhanced 
Critical Incident Technique (ECIT) research method (Butterfield, Borgen, Amund- 
son, & Maglio, 2005) and provides examples from a recent research study that 
utilized the ECIT. Our aim is to respond to requests from students and researchers 
considering utilizing this qualitative research method for details on how to use the 
Critical Incident Technique (CIT) by providing a detailed, practical guide based 
on our own experience using the method. 

Readers who are unfamiliar with the CIT will find detailed descriptions as 
follows: 


266 


L. D. Butterfield, W. A. Borgen, A.-S. T. Maglio, and N. E. Amundson 


1. Flanagan (1954) on the origin and initial uses of this research method. 

2. Woolsey (1986) for its applicability to counselling psychology. 

3. Andersson and Nilsson (1964) and Ronan and Latham (1974) on the 
method’s reliability and validity. 

4. Butterfield et al. (2005) for the CIT’s place in the qualitative research tradi- 
tion, how it has evolved since its origins, the credibility checks that have 
emerged, and suggestions for standardizing use of the method. 

Woolsey (1986) suggested the CIT research method is appropriate for coun- 
selling psychology due to its facility to “encompass factual happenings, qualities 
or attributes, not just critical incidents ... its capacity to explore differences or 
turning points ... its utility as both a foundational/exploratory tool in the early 
stages of research, and its role in building theories or models” (Butterfield et ah, 
2005, p. 480). Many of the studies using the CIT have provided insights into 
psychological processes that yielded clinical tools for counsellors to use (e.g., Mc- 
Cormick, 1997). 

Although the CIT is commonly used in industrial and organizational psychol- 
ogy as well as other fields, this article focuses on utilizing the method in counsel- 
ling psychology research. We begin with a brief discussion of the CIT’s place as a 
qualitative research method, a description of the method, and an overview of the 
enhancements made to the method (resulting in the ECIT method described here). 

THE CRITICAL INCIDENT TECHNIQUE AS A QUALITATIVE RESEARCH METHOD 

Although the CIT arose from the World War II Aviation Psychology Program 
of the US Army Air Forces for selecting and classifying aircrews (Flanagan, 
1954), it has since grown beyond its original application and is now a qualitative 
research method whose influence has expanded into many disciplines including 
counselling, nursing, psychology, education, job analysis, marketing, social work, 
and organizational learning investigating counselling psychology questions (But- 
terfield et ah, 2005; Woolsey, 1986). The CIT as described by Flanagan meets 
the qualitative research descriptions offered by Denzin and Lincoln (1994) and 
Creswell (1998) as “the researcher is the key instrument of data collection; data are 
collected as words through interviewing, participant observation, and/or qualita- 
tive open-ended questions; data analysis is done inductively; and the focus is on 
participants’ perspectives” (Butterfield et al., 2005, p. 482). 

Creswell (1998) further suggests that each qualitative research method has its 
own set of rules that makes it unique, and understanding those rules is essential to 
conducting the research appropriately. Creswell offers five dimensions along which 
qualitative traditions are described in order to ascertain their distinctive features. 
If we apply these dimensions to the CIT, its distinctive features are the following: 

1. Focus is on critical events, incidents, or factors that help promote or detract 
from the effective performance of some activity or the experience of a specific 
situation or event. 
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2. Discipline origin is from industrial and organizational psychology. 

3. Data collection is primarily through interviews, either in person (individually 
or in groups) or via telephone. 

4. Data analysis is conducted by determining the frame of reference, forming 
categories that emerge from the data, and determining the specificity or 
generality of the categories. 

5. Narrative form is that of categories with operational definitions and self- 
descriptive titles. These features are what distinguish the CIT from other 
qualitative methods (Butterfield et al., 2005, p. 483). 

More details about the data analysis in a CIT study is offered in the next sec- 
tion. 


APPLYING THE CIT 

Flanagan (1954) described the CIT as having five major steps: (a) ascertain- 
ing the general aims of the activity to be studied, (b) making plans and setting 
specifications, (c) data collection, (d) data analysis, and (e) data interpretation 
and report on the findings. Data interpretation involves three primary stages: (a) 
determining the frame of reference, (b) forming the categories, and (c) determining 
the level of generality or specificity to be used in reporting the data (Butterfield 
et al., 2005; Flanagan). 

Descriptions in the literature of how to operationalize and apply these compo- 
nents of the method are light on details about how each of these steps is conducted 
and how the categories are formed. In this article, we will highlight a completed 
research project (Butterfield, 2006) that exemplifies the use of the basic CIT proc- 
ess, but also highlights the use of an extensive set of credibility checks developed 
over the past two decades that are intended to increase the trustworthiness of a 
CIT study’s results and are consistent with Flanagan’s original approach (But- 
terfield et al., 2005). 

These nine credibility checks, as well as two other enhancements to Flanagan’s 
(1954) original CIT (the inclusion of contextual questions at the start of the 
research interview in order to provide background information for the CIT data, 
and the inclusion of questions regarding wish list items in addition to those that 
were present that helped or hindered in the situation) (Butterfield & Borgen, 
2005; Kemppainen, O’Brien, & Corpuz, 1998), constitute the enhancements to 
the original CIT that we are now referring to as the ECIT. 

Wish list (WT) items are those people, supports, information, programs, and 
so on, that were not present at the time of the participant’s experience, but that 
those involved believed would have been helpful in the situation being studied. 
In this article we (a) discuss the choice of the original CIT method, (b) describe 
each of the five steps involved in conducting a CIT study as identified by Flana- 
gan, and (c) outline in detail ways to implement the enhancements to the original 
CIT approach. 
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CHOOSING THE CIT 

Each qualitative research method is designed to answer a different kind of 
research question. For example, phenomenology explores a person’s experience of 
something, grounded theory explores the process of something, and a case study 
provides a description of a single case of an individual, event, organization, proc- 
ess, and so on (Creswell, 1998, 2008). 

The CIT explores what helps or hinders in a particular experience or activity 
(Butterfield et ah, 2005). The CIT is exploratory by nature and is appropriate 
to use when the researcher is interested in learning more about little-understood 
events, incidents, factors, or psychological constructs “that help promote or detract 
from effective performance of some activity or the experience of a specific situa- 
tion or event” (Butterfield et ah, p. 483). Our illustrative study involved people 
who described themselves as successfully navigating the changes they encountered 
that affected their work. 

The research question that drove the study was “What helps or hinders work- 
ers who successfully navigate and thrive when faced with changes that affect their 
work?” (Butterfield, 2006, p. 9). The researcher began with the assumption that 
there is a discrete knowledge or experience possessed by these individuals that can 
be elicited through self-reflection of successful strategies used. CIT was appropriate 
in this research context because the knowledge sought was particular to a set of 
individuals (e.g., workers) who had expertise in a particular area (e.g., doing well 
in the face of changes affecting their work). 

THE FIVE STEPS IN CONDUCTING A CIT 

Step 1 : Ascertaining the General Aims of the Activity Being Studied 

Once the researcher has clarified the domain of inquiry, formed the research 
question, and determined that the CIT is the appropriate method, the first step 
in using the CIT, according to Flanagan (1954), is ascertaining the general aims 
of the activity being studied. As stated by Butterfield et al. (2005, p. 478), “un- 
derstanding the general aim of the activity is intended to answer two questions: 
(a) what is the objective of the activity; and (b) what is the person expected to 
accomplish who engages in the activity”? Our study focused on the naturally 
occurring, self-renewing, self-sustaining strategies used by workers who self- 
identified as doing well despite changes affecting their work. Participants were 
selected based on having experienced changes affecting their work, self-reporting 
that they had handled the changes well, being prepared to talk about what had 
helped or hindered their ability to handle the changes well, and being available 
for two interviews. The purpose of the research interview was to elicit the discrete 
strategies of these workers in order to build a composite picture of the approaches 
used by these successful workers that they perceived as helping or hindering them 
in being successful, and those they thought would have been helpful. 


Using the ECIT in Counselling Psychology Research 


269 


Step 2: Making Plans and Setting Specifications 

Once the researcher has determined the general aims of the activities or psy- 
chological processes to be studied, the next step in a CIT study is to make plans 
and set specifications (Flanagan, 1954), which has been described as “(a) defining 
the types of situations to be observed, (b) determining the situation’s relevance to 
the general aim, (c) understanding the extent of the effect the incident has on the 
general aim, and (d) deciding who will make the observations” (Butterfield et ah, 
2005, p. 478). This involves deciding what to observe or ask about; creating an 
interview guide or set of protocols for interviewers to follow; and training people 
in the purpose of the study, the intent of the questions, and how to conduct CIT 
interviews. In essence, this process helps ensure that everyone involved understands 
the aim of the study and follows similar procedures so consistency is maintained 
across researchers involved in conducting the CIT interviews. 

Using an interview guide is a particularly valuable tool for CIT researchers. 
As in other qualitative research studies, it serves as a record of the interview, a 
back-up in case of equipment failure, a way of keeping the interviewer focused 
on the participant’s story, and a resource for referring back to previous comments 
made by the participant that require follow-up. It also serves as a tool to ensure all 
interview guide questions have been asked and responded to. The format of the 
interview guide is important in a CIT study to ensure ease of identifying critical 
incidents (CIs) and WL items, and that the supporting details for each item (an 
example, and the importance of the item for the participant in the experience 
being studied) is captured during the research interview. Earlier CIT studies we 
conducted did not format the interview guide in this way, making it difficult to 
ascertain what data constituted a Cl or WL item. Appendix A provides an example 
of the interview guide format used for our study. 

Step 3: Collecting the Data 

Data collection in a CIT study involving psychological concepts is primarily 
done via interviews, either in person or over the telephone. The major vehicle for 
collecting the CIT data is the first interview. It is important to ensure interview- 
ers are trained to conduct the interviews, obtain informed consent, and address 
any questions participants may have about the interviewer or the research study. 
Although Flanagan (1954) advocated collecting data through expert observation, 
that is often not possible when exploring psychological constructs or experiences 
from the individual’s perspective. 

In our experience, using the CIT in counselling psychology research is most 
effective when data are collected through in-person interviews. Establishing rap- 
port, allowing participants to tell their story during the contextual component, 
and being able to follow up using probes and questions as needed yield rich data 
that would likely not be obtained if other methods were used. 

The CIT interview begins by obtaining the participant’s informed consent and 
answering any initial questions. The first interview with participants is the most 
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important because it (a) allows the participant to tell his or her story and feel heard 
and understood, (b) provides the background information against which the CIs 
and WL items can be understood, (c) elicits the CIT data, and (d) gathers the demo- 
graphic data that describe the sample. In our study, the first interview took an aver- 
age of 2.5 hours. It is critical (a) not to rush this interview, (b) to cover all elements 
in the interview guide, and (c) to ask clarifying or follow-up questions as needed. 

One important objective of CIT interviewing is to explore the same content 
areas at the same level of detail with all participants. In this research project we 
explored factors that helped or hindered participants’ ability to handle change well 
and factors they thought would have been helpful had they been available. This 
does not mean asking exactly the same questions in the same sequence. In some 
cases, eliciting the helping or hindering factors will require little more than asking 
the interview question and letting the participant tell his or her story. In others it 
will require more work in the form of probing, following up on previous remarks 
made by the participant, or asking for clarity, additional details, or examples. 

In our experience, conducting CIT and other qualitative research interviews 
following different methodologies using basic empathy along with other active 
listening skills and being curious while also being respectful greatly facilitates 
the interview process. Whenever possible, it is desirable to ask follow-up ques- 
tions during the first interview regarding points made by the participant that are 
incomplete or unclear, although follow-up questions can be asked during the 
second interview if necessary. It is not necessary to keep a record of every prob- 
ing question or follow-up question used in each interview. The information that 
is elicited is of primary interest; the way it is elicited is handled by an interview 
fidelity check discussed below. 

Flanagan (1954) stated that participant interviews should continue until ex- 
haustiveness or redundancy in the data occurs. This refers to the point at which 
participants mention no new CIs or WL items and no new categories are needed to 
describe the incidents. Exhaustiveness is Flanagan’s term for the concept of satura- 
tion as defined by Creswell (2008). Flanagan and Butterfield et al. (2005) discuss 
exhaustiveness in more detail as a credibility check and it is mentioned again 
below, but it is important to note here that achieving exhaustiveness determines 
when a sufficient number of participants have been interviewed. The researcher 
may choose to continue interviewing participants for other reasons, as was the 
case in our study, but it is not required by the CIT. 

Step 4: Analyzing the Data 

A CIT study typically elicits masses of data, and analyzing it is considered the 
most difficult step involved in conducting a CIT study (Oaklief, 1976; Woolsey, 
1986). In this section we describe how to organize the raw data, extract the CIs 
and WL items, and create the categories that describe them. It presupposes the 
interview recordings have been transcribed and returned to the researcher along 
with transcripts from which the researcher can work. Analysis follows the steps 
prescribed by Flanagan (1954) and Butterfield et al. (2005), namely: 
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1. Determining the frame of reference. This arises from the use that is to be 
made of the data. In our study, we wanted to use the results to create coun- 
selling interventions for people struggling with change and to inform career 
development theories. 

2. Formulating the categories derived from grouping similar incidents (which 
requires experience, judgement, and insight). 

3. Determining the level of specificity or generality to be used in reporting the 
data. This is determined by practical considerations such as project budget, 
number of people available to analyze the data, the extent to which a few 
general behaviours will be useful compared to several dozen specific behav- 
iours, and so on. 

Organizing the raw data. Although there are myriad ways to organize the raw 
data, we have found two to be particularly effective: a manual method and one 
that utilizes a qualitative research data analysis software program. Because the 
procedures used for computer coding are based on established manual procedures, 
the manual procedures are described here. We begin by placing the interview 
transcripts into a three-ring binder with tab dividers separating each interview, 
and investing in a good supply of Post-it notes and as many different coloured 
highlighting markers as are available. The researcher chooses a colour scheme for 
highlighting the various interview guide components in the transcript, such as 
the contextual data and the CIT portion (with different colours for each of the 
helping CIs, hindering CIs, and WL items), as well as the examples and helping 
or hindering qualities of the incident. 

Identifying the CIs and WL items. The first data to be analyzed should be the CIs 
and WT items because, using the ECIT approach, they are subject to an extensive 
array of credibility checks. Extraction of the CIs and WL items is done in batches 
of three randomly chosen interviews following the general guidelines described by 
Flanagan ( 1 954) around sorting a small sample of incidents into piles (categories) 
that relate to the frame of reference, and the more specific standard established by 
Borgen and Amundson (1984). 

Starting with the first interview, the researcher highlights with the appropri- 
ately coloured pen any text that appears to be a helping Cl that is supported by 
examples that describe the incident along with its importance or impact on the 
participant as it relates to the frame of reference of the study. The researcher makes 
note of any items that appear to be helping or hindering CIs or WL items where 
the participant did not describe their importance or impact in order to follow 
up with the participant in the second interview. An example of this would be an 
item in the contextual component of the interview such as “it helped that my 
colleagues were supportive,” which has no supporting details in the transcript and 
is not mentioned again in the critical incident component of the interview. In 
such a case, the researcher would ask the participant during the second interview 
whether this was, in fact, a helping Cl, and if the participant agreed, then he or 
she would be asked to describe its importance and provide an example of a time 
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it helped. Only if the participant agrees that the item is a Cl and provides an ex- 
ample or importance details is the incident included in the final research results. 
This entire process is repeated for the hindering CIs and for the WL items in the 
first, second, then third transcript. At this point the researcher moves to the next 
step with these three interviews: creating the categories. 

Creating the categories. Creating the categories requires the researcher to be 
clear at the outset about how the data will be used: is it for clinical interventions, 
counsellor training, or some other purpose (Butterfield et ah, 2005; Flanagan, 
1954)? The purpose for which the data are being collected influences the forma- 
tion of the categories and their level of specificity. In our study, we wanted to learn 
about the naturally occurring strategies used by people who were doing well with 
changes affecting their work, hoping that the information could be used to assist 
other workers who may not be doing well with such changes. Thus the data were 
to be used primarily to inform therapeutic interventions and training workshops, 
as well as future research. 

Forming categories requires the use of inductive reasoning, patience, and the 
ability to see similarities and differences among the hundreds of CIs and WL 
items provided by participants. The categorization process starts by extracting the 
helping CIs from the first transcript. The researcher can do this either physically 
(e.g., copying highlighted passages onto index cards or cutting the passage out of a 
photocopy of the transcript) or electronically (by creating a text document for each 
participant number with headings for helping CIs, hindering CIs, and WL items). 

This new electronic document has a second purpose: it can later be sent to 
the participant for the crosschecking credibility check described in the following 
section. In order to easily calculate participation rates (also discussed later in this 
article), it is important that the participant number is placed in parentheses at the 
start of each Cl or WL item as it is moved into this document. 

At this point, the researcher’s job is to examine the helping CIs to discern any 
patterns, themes, similarities, or differences among the incidents. In most cases 
there are few similarities across CIs obtained from the same interview, although 
it can happen. In our study, one participant included three helping CIs: “good 
working relationship with your boss,” “working with a coach,” and “keeping in 
contact with friends, social support.” 

After considering these three CIs in the context of the interview, the researcher 
decided they had some form of support in common. Creating a new document 
for the categories now being formed, the researcher copied and pasted these CIs 
into it under the heading “Support.” This process is repeated for the remaining 
helping CIs in the interview. It is common to have nearly as many categories at 
this stage as there are CIs. Once the helping CIs have been placed into categories, 
the process is repeated for the hindering CIs, then again for the WL items. It is 
helpful to create a working table to track the number of new categories created 
when working on each interview; it will aid in determining the point at which 
exhaustiveness occurs. This table is discussed again later in the article. 
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Next the researcher examines the CIs and WL items from the second transcript. 
The researcher again starts by examining the helping CIs, placing any that fit into 
the existing categories, and creates new categories for those that do not fit. This 
process is repeated for the hindering CIs and the WL items from the second tran- 
script. It is common at this point to still be creating additional categories. Once 
all the CIs and WL items from this interview have been placed into existing or 
new categories, the researcher updates the table that details the number of new 
categories created from that interview’s data. This process is repeated again for the 
third transcript of the batch. 

Once the CIs and WL items from the first three transcripts have been cat- 
egorized and the table updated, the researcher begins work on the next batch of 
three transcripts by identifying and extracting the CIs and WL items and repeat- 
ing the process of highlighting them and placing them into existing categories 
or creating new ones. As new CIs and WL items are placed into the categories, it 
is usually necessary to rename the categories and make ongoing decisions about 
the level of specificity to be used for reporting the results. At times, a category 
may become so broad or large that greater specificity is needed in order to create 
a richer understanding of the research area. In the above example of creating the 
category named “Support” to accommodate three helping CIs, the category was 
later divided into three discrete categories: “Support from Friends and Family,” 
“Support from Work Colleagues,” and “Support from Professionals.” 

The researcher needs to consider a number of issues when deciding whether 
to merge small categories together (those with few CIs or WL items) or break 
large categories into smaller ones as just described. One is the use to be made of 
the data: will the change make it easier or harder to use the data for its intended 
purpose? Another is whether there is significant overlap among categories such 
that they are difficult to distinguish from one another. 

Still another consideration is whether 25% of participants identified incidents 
that fit into a particular category, which is the standard established by Borgen and 
Amundson (1984) as the minimum participation rate needed to form a viable 
category. If not, is there another existing category into which these CIs and WL 
items could be placed without compromising the meaning of either category, or 
can a new category be created that would accommodate the smaller ones? 

This process of placing incidents into categories, examining the categories to 
see if they make sense, deciding whether there is significant overlap among cat- 
egories that requires them to be merged, and determining the need to break large 
categories into smaller ones continues until the CIs and WL items from all but 
10% of the interviews have been placed and the category scheme appears to be 
complete. At that point, the self-descriptive titles of the categories are finalized 
and an operational definition is written for each category. 

In our research, the “Self-care” category was a merger of several smaller initial 
categories. The operational definition included the various themes found within 
the broader category: health initiatives and positive, growth/learning, self- affirm- 
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ing, and self-soothing activities that fill their physical, social, emotional, intellec- 
tual, and change processing needs to help them deal well with change. 

Placing the CIs and WL items from the final 10% of interviews into the cat- 
egories that have been developed is the final step in creating a list of categories to 
represent the incidents provided by the participants (Butterfield et ah, 2005). In 
the majority of cases, after this point no new categories are needed and no changes 
to either the category titles or operational definitions are required. Occasionally 
there may be a need to refine either the category title or the operational defini- 
tion, but in our experience this is rare. Once all the categories have been created 
and the operational definitions written, the researcher is ready to move on to the 
next step, which includes conducting the credibility checks as part of the data 
interpretation and reporting process. 

Step 5: Interpreting the Data and Reporting the Results 

Step 5 of a CIT study consists of interpreting the data and reporting the results 
(Flanagan, 1954). It is in this step of Flanagan’s process where we have made ad- 
ditions to the CIT that, in addition to incorporating the contextual questions at 
the beginning of the interview and including the WL items, make it the ECIT. 
These additions to Flanagan’s method of interpreting data and reporting the results 
involve the use of the nine credibility checks. We have found these credibility 
checks to be especially useful when the study involves the report of perceptions 
regarding an experience rather than direct observation of a behaviour. A discus- 
sion of the history and rationale of the nine credibility checks that follow can be 
found in Butterfield et al. (2005); we focus here on the application of these checks. 

Audiotaping interviews. Maxwell (1992) discusses the concept of descriptive 
validity in qualitative research, which concerns the accuracy of the account. But- 
terfield et al. (2005) suggest that audiotaping the data collection interviews allows 
researchers to capture the participants’ words and then work directly from either 
the audiotapes or the transcripts created from the tapes. 

Interview fidelity. Creswell (1998) highlights the importance of following a 
method’s established protocols in order to strengthen the robustness of the find- 
ings. To ensure that (a) the CIT research method is being followed, (b) the inter- 
viewer is not asking leading questions or prompting the participant, and (c) the 
interview guide is being followed, it has become customary to have an expert in the 
CIT research method listen to every third or fourth taped interview (Butterfield 
et ah, 2005). This check can be performed by the researcher’s thesis supervisor, 
another member of the research team, or a third party external to the research 
team who knows the CIT method. Feedback is then provided to the interviewer 
prior to conducting the next interview. 

Independent extraction of CIs. Independent extraction of CIs requires that an 
individual other than the person who initially identified CIs and WL items review 
and independently identify (or extract) what they think are the CIs and WL items. 
Butterfield et al. (2005) suggest it is customary to randomly choose 25% of the 
transcripts to give to this independent individual. This number has evolved based 
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partly on time, cost, and effectiveness considerations, but also because it is consist- 
ent with Andersson and Nilsson’s (1964) process when examining the reliability 
and validity of the CIT (Butterfield et ah). Once the person has extracted what 
he or she thinks are the CIs and WL items, the researcher compares what each has 
extracted and computes the percentage of agreement. For incidents that do not 
match, the researcher and independent reader can discuss the discrepancy for the 
purpose of resolving the difference. If no resolution can be made, the concordance 
rate would reflect less than a 1 00% match, and the incident in question would 
not be used in further analysis. 

Exhaustiveness. As mentioned above, to determine the point at which exhaus- 
tiveness occurs, it is necessary to keep a log of each interview as its CIs and WL 
items are placed into the category scheme being created. As already indicated, we 
found it was helpful to create a table with the date the interview was coded in the 
far left column, the participant number in the next column, the date the CIs and 
WL items from the interview were placed into the category scheme in the next 
column, and in the far right column a notation about whether any new helping, 
hindering, or WL categories were created to accommodate the data. An example 
of the table is attached as Appendix B. 

In our research, a glance at the table told us that for the first three interviews all 
new categories were created; for the next two interviews 12 new helping catego- 
ries, 3 hindering categories, and 4 WL categories emerged; and no new categories 
emerged for the sixth through twentieth interviews. Exhaustiveness was therefore 
achieved after the fifth interview. As mentioned above, interviewing may continue 
beyond this point for a number of reasons, including the situation in this case 
where interviews had already been scheduled with the remaining participants. 

Participation rates. Reporting participation rates is important when establish- 
ing the credibility of categories as they are being formed, and for assessing the 
relative strength of a category when reviewing the results of the study (Borgen & 
Amundson, 1984). In order to calculate participation rates, the researcher must 
include the participant number with every Cl or WL item that is copied into the 
“Categories” document discussed above. By doing so, it is possible to simply count 
up the number of different participant numbers under each category and divide 
that number by the total number of participants to determine the participation 
rate. In our study, under the helping category “Support from Friends & Family,” 
15 of the 20 participants provided items for this category, which translates into a 
participation rate of 75%. 

Placing incidents into categories by an independent judge. The purpose of this 
step is to have an independent person place 25% of the CIs and WL items into 
the categories that have been created by the researcher and to calculate the match 
rate between that person’s placements and those of the researcher (Butterfield et 
al., 2005). The researcher randomly chooses 25% of the incidents within each 
category and sends them to an independent judge, along with the category head- 
ings and operational definitions, with instructions to place each incident into 
the category where the judge thinks it belongs. The researcher then compares the 
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judge’s placement of CIs and WL items into categories with the researcher’s own 
placement. Andersson and Nilsson (1964) suggested a match rate guideline of 
80% or better for this credibility check. 

In the event of a discrepancy, a decision needs to be made about how to rec- 
oncile the differences. The rule we have used, which is effective and respectful of 
participants’ experience, is to base the final decision about the category into which 
an incident is to be placed on the participant’s response during the participant 
cross-checking, which is discussed next. 

Cross-checking by participants. The second interview with participants is con- 
ducted after all the CIT data from the first interviews have been coded, analyzed, 
and placed into categories with operational definitions, and self-descriptive titles 
have been created by the researcher. It also follows the independent extraction of 
CIs and placement of the incidents into the category scheme by an independent 
judge, and the achievement of acceptable match rates. 

The second interview is conducted for a number of reasons. First, it invites 
participants to review the interpretations made by the researcher to ensure they 
are supported by study participants, thus confirming the CIs and WT items 
extracted from the transcripts. Second, it affords participants an opportunity to 
review the categories into which their CIs and WL items have been placed and 
to comment on how well the category titles capture their lived experience of the 
specific incidents. Third, it confirms whether or not the CIs and WL items have 
been placed into appropriate categories. Finally, it ensures participants’ voices have 
been honoured and accurately reported. 

Unlike the first interview that is conducted in person, the second interview is 
generally conducted by e-mail and telephone. It is not necessarily taped, and it is 
not transcribed. The second interview is a credibility check with three parts: (a) 
getting input on the CIs and WL items extracted from the participant’s first inter- 
view, (b) obtaining feedback regarding the categories into which they have been 
placed, and (c) following up on questions arising from analyzing the participant’s 
data from the first interview. 

The first part entails sending to the participant via e-mail the list of CIs and 
WL items extracted from their first interview, and a separate list of the categories 
with their CIs and WL items placed into them. The participant is asked to review 
the list of CIs and then answer the following questions: 

1 . Are the helping/hindering CIs and WL items correct? 

2. Is anything missing? 

3. Is there anything that needs revising? 

4. Do you have any other comments? 

It has been our experience that the majority of participants agree with the CIs 
and WL items as listed. When that happens, no further action is needed and the 
researcher can move on to the second part of the interview. If the participant asks 
for clarification of any Cl or WL item, it is appropriate to discuss the context of 
the incident with the participant by reading the transcript excerpt from which the 
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item was taken. We have found that providing this information jogs the person’s 
memory and usually results in their agreeing with the item. If the participant 
disagrees with an item after the context is provided, he or she has the final say 
about whether it should be included. 

The participant is then asked to review the categories into which the CIs and 
WL items have been placed and to answer the following questions: 

1 . Do the category headings make sense to you? 

2. Do the category headings capture your experience and the meaning that the 
incident or factor had for you? 

3. Are there any incidents in the categories that do not appear to ht from your 
perspective? If so, where do you think they belong? 

The researcher and participant discuss any potential changes suggested by the 
participant. Although generally the participant’s wishes are honoured and the 
changes made, if the researcher disagrees with the participant’s wishes it is permis- 
sible to discuss the concerns and arrive at a mutually agreeable solution. 

The last part of this second interview consists of following up on questions 
that arose from the participant’s first interview. This is most often needed when 
something in the first interview sounded like it was a Cl or WL item, but the 
supporting importance or example information was missing. The second interview 
offers a chance to ask the participant whether an item is a helping, hindering, or 
WL item and, if the person agrees it is, to ask them for the supporting details. 

Assuming the supporting details are provided, the item is then coded, added to 
the total number of helping, hindering, or WL items, and added to the appropri- 
ate category. The researcher would advise the participant, during the telephone 
conversation, of the category into which the incident will be placed to obtain the 
participant’s agreement. 

We have generally found it to be most productive to first e-mail the Cl and 
WL item lists along with the category headings into which they have been placed, 
then follow up with a telephone conversation that allows the researcher to ask for 
feedback, probe for more information, and deal with any questions the participant 
may have. The e-mail provides an audit trail of the interaction; the telephone call 
allows the researcher to ask the follow-up questions. At the start of the telephone 
conversation it is important to advise the participant that the researcher will be 
taking notes of what is discussed and will e-mail them to the participant at the 
end of the conversation to review, approve or revise, and return via e-mail to the 
researcher. 

Expert opinions. The eighth credibility check consists of submitting the catego- 
ries that have been created to two or more experts in the field. This occurs once the 
category scheme is finalized, which is usually after the participant cross-checking 
has been completed. When choosing the experts, it is important that they have 
knowledge of the area being studied, even if they are not familiar with the CIT. 
The experts are asked to provide responses to three broad questions after review- 
ing the categories: 
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1. Do you find the categories to be useful? 

2. Are you surprised by any of the categories? 

3. Do you think there is anything missing based on your experience (Butterfield 
et ah, 2005; Flanagan, 1954)? 

In our research, we submitted the category scheme to a psychologist with ex- 
tensive business experience in the areas of career management and organizational 
consulting, and to a human resource professional and management consultant with 
more than 25 years of experience. Since both experts confirmed all the categories 
were congruent with their knowledge of the research and professional field and 
both believed the categories were useful, we were then ready to move on to the 
final credibility check. 

Theoretical agreement. As described by Butterfield et al. (2005) and based on 
Maxwell’s description (1992), theoretical agreement has two parts. The first has to 
do with articulating and reporting the assumptions underlying the study. In our 
study, some of the assumptions were (a) people experience change, are aware of 
it, and are able to describe their responses; (b) change has increased in all arenas of 
people’s lives; and (c) the environment is no longer stable, and therefore it is not ap- 
propriate to assume that the individual who is experiencing difficulties dealing with 
change is at “fault.” The researcher then reviews the scholarly literature in appropri- 
ate disciplines to see if the assumptions are supported. In our research, we reviewed 
the literature in workplace change, social psychology, psychological thriving, busi- 
ness, stress and coping, positive psychology, career counselling, vocational psychol- 
ogy, and other relevant areas, finding support for these underlying assumptions. 

The second part of theoretical agreement compares the emergent categories 
with relevant scholarly literature. When this literature check was conducted for 
our study, support for all 1 0 categories that had been developed was found in the 
literature. It is important, however, to remember that by definition a CIT study is 
exploratory. Lack of support for one or more categories in the literature may mean 
that the study has uncovered something new that is not yet known to researchers 
rather than suggesting a category is not sound (Butterfield et al., 2005). It may 
also indicate that different areas of theory and research than those that have been 
used up to now are needed to describe or explain the area that has been studied. In 
either case, these categories are considered to be in need of more study to further 
support or challenge them. 

Reporting the results. The final component of Step 5 in a CIT study is to report 
the results (Flanagan, 1954). Flanagan considered this to be one of the researcher’s 
primary responsibilities, which can be discharged any number of ways. Most often 
it is accomplished through publishing articles or chapters in relevant scholarly or 
trade journals or books, or through presentations at trade or scholarly conferences. 

CONCLUDING REMARKS 

The CIT has proven to be a versatile, adaptive, and robust research method 
that will likely continue to evolve as researchers utilize it to shed light on new and 
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innovative research questions. Our intention in writing this article was to illustrate 
the process involved in conducting an ECIT study by providing a detailed guide 
based on our own experience with the approach. We hope this elaboration will be 
of assistance to students and researchers unfamiliar with the CIT research method 
and the ways it has evolved over the past several years. 

One final word: although the number of credibility checks may appear daunt- 
ing, they can be conducted fairly quickly and they get easier as familiarity with 
them increases. The payoff is having followed a well-documented research process 
that increases the researcher’s confidence in the credibility and trustworthiness of 
the results obtained from the study. 
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APPENDIX A 

Sample Interview Guide: “Doing Well” Strategies Being Used 

Participant #: Date: 

Interview Start Time: 
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1. Contextual Component 

Preamble: As you know, I am investigating the ways in which working 
women and men have successfully handled change(s) that affect their work. 
This is the first of two interviews, and its purpose is to collect information 
about the changes you have experienced and the ways in which you are 
dealing well with them. 

a. As a way of getting started, perhaps you could tell me a little bit about 
your work situation. 

b. You volunteered to participate in this study because you identified 
yourself as experiencing workplace change and doing well with it. What 
does “doing well” mean to you? 

c. What are the changes that have affected your work life? 

d. How have these changes affected your work life? (Probe, as needed: Are 
there any other impacts on your work?) 

2. Critical Incident Component 

Transition to Critical Incident questions: You said that even with all these 
changes, you rated yourself as a 5-6 (or whatever the participant rated him- 
or herself in question 1 (c) above). 

a. What has helped you in doing well with the changes that have affected 
your work? (Probes: What was the incident/factor? How did it impact 
you? - e.g.: “Persistence is helping. How is it helping?” Can you give 
me a specific example where persistence helped? How did that help you 
to do well in handling the changes affecting your work?) 


Helpful Factor & 
What It Means to 
Participant (What 
do you mean by 
...?) 

Importance (How 
did it help? Tell me 
what it was about 
. . . that you find so 
helpful.) 

Example (What led 
up to it? Incident. 
Outcome of 
incident.) 








b. Are there things that have made it more difficult for you to do well? 
(Alternative question: What kinds of things have happened that made 
it harder for you to do well?) 


Hindering Factor & 
What It Means to 
Participant (What 
do you mean by 
...?) 

Importance (How 
did it hinder? Tell 
me what it was 
about . . . that you 
find so unhelpful.) 

Example (What led 
up to it? Incident. 
Outcome of 
incident.) 
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c. Summarize what has been discussed up to this point with the participant 
as a transition to the next question: 

We’ve talked about what’s helped you to do well (name them), and 
some things that have made it more difficult for you to do well (name 
them). Are there other things that would help you to continue doing 
well? (Alternative question: I wonder what else might be helpful to you 
that you haven’t had access to?) 


Wish List Item & 
What it Means to 
Participant (What 
do you mean by 
...?) 

Importance (How 
would it help? Tell 
me what it is about 
... that you would 
find so helpful.) 

Example (In what 
circumstances might 
this be helpful?) 








d. Have you always handled change well? 

(Circle one) Yes No 

e. If not, when did this change for you? 

£ What happened that caused you to begin handling change well? 

3. Demographics Component 

i. Occupation 

ii. Number of years in this occupation 

iii. Occupation/job level 

iv. Length of time in current job 

v. Industry in which the person works 

vi. Number of years in this industry 

vii. Length of service in this company 

viii. Age 

ix. Sex 

x. Income level (household) 

xi. Country of birth 

If not Canada, (a) length of time in Canada; and (b) 1st language 

xii. Marital status 

xiii. Family status/parental status 

xiv. Education level 


Interview End Time: 
Length of interview: 
Interviewer’s Name: 
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APPENDIX B 

Sample Table for Tracking the Emergence of New Categories 


Date of CI/WL 
Extraction 

Participant # 

Date Categorized 

New Categories Emerged? 

February 14 

13 

February 28 

All new categories emerged 

February 18 

39 

February 28 

February 25 

54 

February 28 






March 2 

14 

March 15 

12 new HE; 3 new HI; 4 new 
WL categories emerged 

March 8 

15 

March 15 

March 12 

51 

March 15 

No new categories emerged 





March 17 

16 

March 25 

No new categories emerged 

March 20 

32 

March 25 

No new categories emerged 

March 24 

40 

March 25 

No new categories emerged 





March 26 

52 

March 29 

No new categories emerged 

March 27 

35 

March 29 

No new categories emerged 

March 28 

65 

March 29 

No new categories emerged 





March 30 

60 

April 3 

No new categories emerged 

March 31 

42 

April 3 

No new categories emerged 

April 2 

64 

April 3 

No new categories emerged 





April 5 

41 

April 10 

No new categories emerged 

April 7 

38 

April 10 

No new categories emerged 

April 9 

33 

April 10 

No new categories emerged 





April 13 

21 (10% left to 
end) 

April 15 

No new categories emerged 

April 14 

37 (10% left to 
end) 

April 15 

No new categories emerged 


Note: HE = Helpful Critical Incidents; HI = Hindering Critical Incidents; WL = Wish List items 
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